A Performance Test Platform

We implemented a fully automated benchmarking system, the performance test platform (PTP). It is developed with special emphasis on flexibility and therefore uses library centric application design combining several orthogonal modules. The modules consist of a base of Bash shell scripts and an orthogonal web front-end written in PHP. We thereby ensure that no restrictions are imposed on the benchmarked systems. Furthermore we show the platform’s application and results obtained by benchmarking different algebraic equation specification approaches. INTRODUCTION AND MOTIVATION We present the performance test platform (PTP), a fully automated benchmark system to consequently analyze compiletime and run-time performance of different algorithms, programming paradigms, or even complete applications. The compiler attributes are fully parametrized, which allows the use of different compilers and options on multiple hosts with several operating systems. It is not only possible to pass various options to the compiler, but it is also possible to pass different options to the newly compiled executable. The rapid development of micro processors and the huge range of currently available processor types requires very complex tools to analyze applications in order to optimize their run-time performance. Additionally, compilers also undergo their own evolution and different compilers introduce a new set of optimization possibilities to control the new features of modern micro processors with each new generation. The possibilities of combining all of these switches and options is increasing even more rapidly and results in a staggering amount of possibilities to choose from. A manual selection of settings to obtain optimal performance is therefore not an efficient task as even different types of applications require a different set of options and switches. A project was therefore initiated with the goal to determine which compiler-options result in optimal code on different architectures, operating systems, and from different programming paradigms. During the development of a batch-jobsystem using Bash shell scripts [12] the idea to extend this to a complete performance test platform was born. Once a system consisting of several shell scripts and a hierarchical directory structure was developed, usability had to be increased to make this system accessible on various platforms without manual interaction. A database system, a graphical user interface, as well as graphical statistic modules have therefore been developed. THE DEVELOPED PLATFORM The base of the developed system consists of a collection of Bash shell scripts and an orthogonally implemented graphical web front-end in the dynamic web programming language PHP [5], cf. Figure 1. Both are supplemented by a MySQLdatabase [13] and GNU R [7], a statistical and graphical suite used to process the results and create graphical output. Figure 1: Overview of the performance test platform. We chose a shell-script backend to ensure the possibility to easily change parameters, templates and scripts manually or by the graphical user interface. We thereby achieve a maximum of freedom and openness for a large variety of systems, languages, compilers, and algorithms to test. The database was implemented in MySQL, because it is widely available, easy to use, and very stable when using InnoDB-engines [15, 11] with foreign constraints. Using these features and a